Evaluating HeLI with non-linear mappings

Peer reviewe

Evaluation of language identification methods using 285 languages

Peer reviewe

Discriminating between Mandarin Chinese and Swiss-German varieties using adaptive language models

Peer reviewe

Language and Dialect Identification of Cuneiform Texts

Peer reviewe

Language Set Identification in Noisy Synthetic Multilingual Documents

Proceeding volume: Part IIn this paper, we reconsider the problem of language identification of multilingual documents. Automated language identification algorithms have been improving steadily from the seventies until recent years. The current state-of-the-art language identifiers are quite efficient even with only a few characters and this gives us enough reason to again evaluate the possibility to use existing language identifiers for monolingual text to detect the language set of a multilingual document. We are using a previously developed language identifier for monolingual documents with the multilingual documents from the WikipediaMulti dataset published in a recent study. Our method outperforms previous methods tested with the same data, achieving an F 1-score of 97.6 when classifying between 44 languages.Peer reviewe

HeLI, a Word-Based Backoff Method for Language Identification

In this paper we describe the Helsinki language identification method, HeLI, and the resources we created for and used in the 3rd edition of the Discriminating between Similar Languages (DSL) shared task, which was organized as part of the VarDial 2016 workshop. The shared task comprised of a total of 8 tracks, of which we participated in 7. The shared task had a record number of participants, with 17 teams providing results for the closed track of the test set A. Our system reached the 2nd position in 4 tracks (A closed and open, B1 open and B2 open) and in this paper we are focusing on the methods and data used for those tracks. We describe our word-based back-off method in mathematical notation. We also describe how we selected the corpus we used in the open tracks.Peer reviewe

HeLI-based Experiments in Swiss German Dialect Identification

Peer reviewe

HeLI-based Experiments in Discriminating Between Dutch and Flemish Subtitles

Peer reviewe

Iterative Language Model Adaptation for Indo-Aryan Language Identification

Peer reviewe

Semantic Domains in Akkadian Text

The article examines the possibilities offered by language technology for analyzing semantic fields in Akkadian. The corpus of data for our research group is the existing electronic corpora, Open richly annotated cuneiform corpus (ORACC). In addition to more traditional Assyriological methods, the article explores two language technological methods: Pointwise mutual information (PMI) and Word2vec.Peer reviewe